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ABSTRACT 

An experiment investigated the effects of the 
instruction rubric at the beginning of an examination paper and of 
the format of the examination questions themselves. A history test 
whose questions originally differed widely in format was modified so 
that its rubric was as clear and concise as possible and all the 
short answer questions were of the same format. The design of the 
experiment produced four versions combining the new and old rubrics 
with the new and old questions so that the effects of the different 
changes would not be confused. Statistical analysis of data from 200 
subjects seemed to indicate that the original version produced the 
highest candidate achievement scores; however, closer study revealed 
that the new rubric was in fact leading to better achievement results 
than the original one. Although the original rubric was clearly 
superior in the familiar situation, when the situation was less 
familiar, the new rubric began to show its worth. Readability 
measures also showed that the new rubric was simpler and easier to 
understand. Analysis of candidates' choice of questions revealed that 
question-style may outweigh the actual subject content in the 
candidate's process of choice and that questions appearing earlier in 
an examination are more likely to be chosen. These findings indicate 
that exam writers should consider the effect of the way a question is 
couched and the importance of readability of instructions. (JL) 
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Problems about the unfairness introduced into examinations by 

question choice have led to the arrangement of examination papers 

becoming increasingly complicated. In turn, the instructions for 

choosing certain numbers of questions from different sections of the 



and 



Che present experiment was designed to investigate the effects o: 
the instruction rubric at the beginning of an examination paper, 
Df the format of the examination questions themselves. 

The findings do appear to hold implications for the constructors of 
examinations. Firstly, if papers are to permit choice and various 
different combinations of questions, then the style and difficulty 
of those questions ought to be comparable. Secondly, if there is to 
be choice, the instructions for that choice should be clear. To 
this end, it would seem reasonable to make use of the techniques 
associated with improving the Readability of material. 

The overall suggestion is made that the needs of the candidate, and 
the possible effects of the content and structure of the examination 
paper upon him, be considered even more actively in the drafting of 
examination papers - in all subjects and at all levels. 

The Current Situation 

Few people would probably disagree with the viewpoint that the purpose of an 
examination is to give the candidate a fair opportunity to demonstrate the 
degree of knowledge and/or skill he has acquired in a particular area. It 
therefore follows that he should not be hampered in that endeavour by the 
examination paper itself. However, problems about the unfairness introduced 
into examinations by question choice have led to the arrangement of 
examination papers becoming complicated. In turn, the instructions for 
choosing certain numbers of questions from different sections of the paper 



have tended to become more complex, creating further difficulties. 
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(j\ A search of the literature reveals considerable evidence that such problems 
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do occur at ail levels in the British examination system: from C.S.E. and 

G.C.E.,* at age sixteen, right up to degree level. There appear to be five 

major areas of difficuJ.ty, which are all inter-connected. They concern: 

The language of the examination 
Question choice 

Differences in intrinsic difficxilty of questions 
The difficulty of rubrics 
Candidates ' anxiety 

Candidates may well not understand the language in which the examination is 
couched. As a result they may also have difficulty in following the 
instructions laid out in the examination rubric and make incorrect or unwise 
decisions in choosing the questions to answer. Furthermore, they may 
unwittingly be putting themselves at a disadvantage by choosing particular 
questions, some of which may well be more difficult than others in the first 
place. 

The need for a closer study of the place of language in education has indeed 
been advocated for some time, and the problem was highlighted by Johnstone and 
Cassels in 1978,"^ reporting an experiment with UOOO secondary school pupils in 
England and Scotland. They experimented with school science examination 
questions, re-vriting them to make them less complex. Pupils' scores were 
found to improve markedly as a result of these alterations, suggesting that 
the question of the appropriateness of the language of examination papers is 
one that should be kept closely in mind when papers are constructed. 
However, interestingly, it is the matter of question choice that has exercised 
researchers most in the last few years. 

How good are candidates at choosing the questions they can answer best? An 

2 

enquiry by Cowan, involving engineering students eind published in 1972, 
would suggest that they are not good, either at selecting the best questions 
to answer or at choosing their best answers once they have completed the paper. 



^ * Certificate of Secondary Education and General Certificate of Education. 
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In addition, there are several pieces of work which demonstrate that choosing 
the best answers is not the most crucial problem, but rather the ways in 
which choice of particular questions may facilitate or handicap the 
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student's performance. For example, Willmott and Hall's finding (1975) 
that candidates tend to answer questions in the order they appear on the 
examination paper, is supported by that of Francis and Owen in 1978 • 

The problem of question choice, in itself serious, is however likely to be 
compounded if individual questions themselves differ in intrinsic 
difficulty, and this is perhaps one of the areas that should be regarded 
with particular concern. We may 'not be able to ensure that candidates will 
always choose the most appropriate questions, but- the construction of a good 
examination should make certain that the questions themselves will not 
produce problems of inequality. 

Wilson, in Northern Ireland, found in both the 1973 and 197^ G.C.E. 'Advanced 
level Physics examinations (taken at age eighteen) that choice accounted for 
possible maximum differences of from k3 to 59 marks, 11 - 15^ of the total 
available.^ 

Even at first degree level, Gowenlock, Mcintosh and Mackaill (l972) found 

similar problems with the 1968 and 1969 final degree examinations in 

Chemistry at Heriot-Watt University. Analyses of these results showed: 

that the questions considered were not of equal difficulty, did 
not test objectives equally and did i^ot discriminate equally.^ 

Their main concern, however, was not the differential difficulty of questions 

but that, dependilig on the choice of questions, the objectives tested by the 

same examination could vary from student to student. The solution they put 

forward was aimed at a more uniform selection of objectives, by dividing the 

paper into several sections, each containing questions with identical 

objectives. Each question would be compulsory but would have alternative 

g 

elements from which the candidate would choose. The same idea was also put 
forward by Willmott in 1972 as a way out of the dilemma caused by the 



differential difficulty of questions. He also suggested that the questions 
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be presented in sections, the choice being limited within each section. . 
This does indeed seem to be a very useful way out of a difficult dilemma, but 
it brings, in turn, its own problems. 

Although helping to resolve the matter of question choice, a more 

complicated arrangement of the examination paper will necessitate a more 

complex rubric, which may. itself produce difficulties for the candidate, . as 

Willmott and Hall themselves point out at the beginning of their book on 

Question Choice. Referring to one particidar paper in Geography they say: 

With papers such as the latter, it may be thought that the 
examination questions themselves were easy to answer compared with 
the disentangling of the rubric; little evidence in fact was 
forthcoming on this point, although it is hardly likely that the 
candidates' state of mind would be improved by such rubrics. 

Although Willmott and Hall leave the problems of the rubric at this point in 
order to go on to the matter of question choice, they have in fact touched 
upon the crucial issues in rubric design; firstly, whether candidates can 
understand from it what they are intended to do, and secondly, how the rubric 
makes them feel about answering the paper: might it inhibit performance? 

k 

Francis and Owen's interviews of chemistry candidates suggested to them 
that there were three main categories of strategies* used in the selection of 
examination questions, and that use of the different strategies was related 
to the candidates' level of anxiety. 

If, as the writer believes, it is the rubric which may ultimately hold the 
answer to the problem of question choice, then it is with the raising of the 
issue of anxiety that one comes to the heart of the matter. 

Several pieces of research have demonstrated that question choice and 
examination performance are affected by the candidate's level of anxiety. 
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* These were 'Planners', 'Rankers' and 'Rushers '"^"^ 



For example, in 1952. S.B. Sarason. Handler and Craighill found that by 

manipulating the testing situation they could alter test performance for 

candidates showing high or low anxiety, and following their train of thought, 

Smith and Rockett (l958)^^ found that it was possible to relieve anxiety by 

the manipulation of test instructions. Also in 1958. I.G. Sarason found that 

under stressful instructional conditions, 'low anxious' candidates performed 

better than 'high anxious' ones, yet when the stress was removed from the 

lit 

instructional situation, there were no differences. 

Given these findings, the logical extension of the argument would be that if 
one could make the instructions, i.e. the rubric, less stress-inducing, 
candidates' anxiety would tend to be relieved and performance would improve. 

What makes a rubric stressful? Hambleton and Traub (l97M investigated the 
effect of item order on performance and on stress in a mathematics test.^^ 
If the test began with difficult items and went on to easier questions, 
performance was significantly poorer and stress was greater. Furthermore, 
experimental work on item-order by Munz and Jacobs, in 1971, suggests that 
examinees given papers beginning with the easier questions have more positive 
feelings after the examination (that it was easier and fairer) than 
candidates given questions beginning with the more difficult ones, or 
questions given in random order of difficulty. 

If starting an examination with difficult questions is harmful, then it is 
not unreasonable to assume that starting with difficult instructions will also 
be harmful. It seems clear that what is important is not whether it is 
questions or instructions that are involved, but rather that starting with a 
difficult situation produces higher anxiety and poorer performance. 

.The danger is that more complicated instructions are generating rubrics which 
are in themselves harder to understand. This produces two main effects. 



Er|c 5 ft 



Firstly, it in6rease8 the stress on the candidate, and secondly, it means 
that the candidate may choose questions unwisely. There is evidence that 
both effects vill in turn adversely .affect performance; and the problem vill 
be compounded if the questions themselves differ in intrinsic difficulty. 

The investigation described in this article involved incorporating different 
rubrics and different question formats into four experimental examination 
papers, to see whether they would affect differentially the candidates' 
performance in the examination. The fact that the examination chosen as the 
basis for the experiment was the East Midland Regional Examination Board's 
Certificate of Secondary Education History examination should not however be 
taken to mean' that the findings apply only to the British situation, nor 
particularly to History, nor even to the school-leaving age-group alone. 

A description of the experiment follows in the next section of this article, 
but the conclusions' outlined :ln the final section are, it is felt, relevant to 
examinations as a vhole, irrespective of specific subject or environment. 

The Experiment * 

The paper chosen as the basis for the experiment was the 1978 C.S.E. History 
Mode 1 Syllabus C paperf—' World Affairs - 20th Century'. The purpose was to 
investigate 

1) the instruction rubric at the beginning of the paper 

2) the format of the questions themselves 

by producing alternative versions of it where the rubric and/or the questions 
were modified in order to facilitate the candidate's task in navigating the 
paper itself.** The- main rubric for the whole, paper was manipulated; but only 

the twelve questions in Part One, and not the six essay questions in Part Two, 

were modified in format. 

•The purpose of revising the rubric was to render it as clear and concise as 



» A fiai account of this experiment may be found in EXPLORING THE FORMAT : the 
19T8 History Examination (University of Leicester School of Educatxon, 
EMREB C.S.E. Research Project Report No. h, April I98O). 

** See Appendix 1 for the rubrics and a sample question. 
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possible, and that of altering the format of the questions to a single comon 
one was to ensure .that candidates were not penalised because of their choice^ 
of questions. On the original History paper, the formats of different 
questions differed videly. They ranged from multiple^hoice to writing 
paragraphs on specific topics,, from one-word answers to interpreting 
historical maps, from filling in slots in given passages to explaining 
quotations. Clearly some of these tasks were likely to be more taxing than 
others, irrespective of the subject titter involved. It was hoped that a 
common format for all the questions in Part One would remove .this source of 
inequality, so that pupiliJ would not be disadvantaged by their .choice of question. 
The essay questions in Part Two of the, paper would remain constant throughout. 

The aim of the experiment was to compare the achievement of candidates using 
papers with the modified 'new'', rubric with that of those using papers with the 
original rubric; and to compare their performance using the new-style questions 
with that of pupils using the original questions. 

The design of the experiment would have to permit this in such a way that any 
differences accruing from the change in rubric would not be confused by the 
modification of the queaUon-atyle , or vice versa. In order to achieve this 
end, four versions of the examination paper were prepared, which would allow 
the comparison of the new and the original rubrics, and of the new-style and 
original questions, both together and separately (where the other elements 
would be held constant). 

The content of these papers was: 

I ORIGINAL RUBRIC + ORIGINAL QUESTIONS 

II NEW RUBRIC + ORIGINAL QUESTIONS 

III ORIGINAL RUBRIC + NEW-STYLE QUESTIONS 

IV NEW RUBRIC + NEW-STYLE QUESTIONS 

and the comparisons would be made as follows: 

a) In order to compare the scores of candidates taking papers containing the 
old rubric with the scores of those taking papers containing the new 
revised one, one would compare results as shown overleaf: 



PAPER I ( ORIGINAL RUBRIC + ORIGINAL QUESTIONS) 

and 

PAPER II (NEW RUBRIC + ORIGINAL QUESTIONS) 



Also: 



PAPER III ( ORIGINAL RUBRIC + NEW-STYLE QUESTIONS) 

and 

PAPER IV ( NEW RUBRIC > NEW-STYLE QUESTIONS) 

"b) To compare the achievement scores of candidates taking papers containing 

questions in the original style of format with those of candidates taking 

papers containing new-style questions, the results of the following papers 

would be compared: 

PAPER I (ORIGINAL RUBRIC + ORIGINAL QUESTIONS ) 

and 

PAPER III (ORIGINAL RUBRIC + NEW-STYLE QUESTIONS ) 



Also : 



PAPER II (NEW RUBRIC + ORIGINAL QUESTIONS ) 

and 

PAPER IV (NEW RUBRIC + NEW-STYLE QUESTIONS ) 



In addition to these two main areas of investigation,- different combinations 
of contrasts of the papers would allow study of other effects of the 
experimental materials. 

c) Thus, to investigate the possible effect of the different rubrics aomhined 
with the different question formats, one would compare the results from 
Paper I with those from Paper IV, 

d) Furthermore, one would be able to see the relative influence, if any, of 
the different rubrics on the effect of question style, by comparing the 
comparison of papers I and III with the comparison of Papers II and IV 
(see (b) above). 

e) The relative influences of the different question-styles on the effect of 
rubric would be visible by comparing the comparison of Papers I and II with 
the comparison of Papers III and IV (see (a) above). 

Over 200 candidates at two schools sat the experimental papers ir.der 
examination conditions, with several control measures in force to ensure the 



internal validity of the experiment. For example, the four different 
examination papers vere packed in rotation (l, II, III| IV) to be given to 
the schools. These papers were then to be laid out in stri.ct order so that 
pupils, given a free choice of where to sit, would be assigned to their 
examination paper completely at random.* 

Secondly, all the candidates were to be warned that there were slight 
variations in the presentation of the examination papers, but that the content 
was exactly the same. One was to oe careful, however, not to let them 
realise that some papers were experimental or novel, so that the results 
would not be biased in favour of the new experimental material, through the 
operation of the "Hawthorne Effect'.** 

In order to avoid confusion for the students about the style of examination 
paper they would in fact do in the summer for the official examination, one 
would suggest to the schools that the students be shown the different papers 
at the end of the examination-experiment. At this point differences could 
even be discussed. In this wr.y one would not interfere with the running of 
the experiment, but would also not confuse the pupils with differences in the 
style of this paper and the summer paper. 

Thirdly, before the experimental session, standard examination procedure 
instructions would be given bo the participating schools, together with the 
examination papers, to "make certain that the procedures were both correct and 
the same for both schools. 

The completed examination papers were subsequently marked twice over, by two 



* The use of random selection meant that the experimental design would 
conform to Campbell and Stanley's requirements-^'^ for True Experimental 
Design No. U, perhaps the most respected design of all, where internal 
validity is concerned. 

The phenomenon by which a subject's performance tends to improve if 
he knows that he is being studied. 

10 
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independent experienced examiners, and the results were analysed using Analysis 
of Variance statistical techniques. As .the manipulations of the rubric and of 
the question format were directed- primarily towards Part One of the paper, and in 
view of the largely statistically non-significant results for Part Two and the 
Whole Examination, the results of the comparisons of the experimental papers 
will deal with Part One only. (Por detailed resuixs of comparisons (a) to Ce) 
see Appendix 2). 

The comparisons of the different papers .showed Paper I (ORIGINAL RUBRIC + 
ORIGINAL QUESTIONS) to lead to the highest candidate achievement scores, and 
Paper III (ORIGINAL RUBRIC + NEW-STYLE QUESTIONS) to .the lowest. Overall, 
Paper I was also seen to result in statistically significantly higher scores than 
Paper II (NEW RUBRIC + ORIGINAL QUESTIONS). The allroriginal Paper I also tended 
to produce better scores. than the all new Paper IV (NEW RUBRIC + NEW-STYLE 
QUESTIONS) . 

One was thus led to the immediate conclusion that the new material was, in 
practice, either less successful than the original material, or at least much the 
same in effectiveness. However, closer study revealed an interesting phenomenon. 
At first sight, the original rubric had appeared superior in a straight 
comparison with the new one, and the traditional paper was superior to the all- 
new one in terms of achievement results. The picture changed, however, when one 
took into account the questions with which the change in rubric was linked. 
Studying the effect of the rubric when it was connected with the traditional 
questions, and its effect when connected with the new-style questions,* revealed 
that the new rubric was in fact leading to better achievement results than the 
original one. When the original questions were involved (Papers I and II) the 
mean score for Paper I, using the original rubric, was several marks better than 
that for Paper II (new rubric). However, when the new-style questions were part 
of the papers (Papers III and IV) it was the mean score for Paper IV, with the 
new rubric, which was superior to Paper III (original rubric). It woulLd appear 



* i.e. comparing: 

I (ORIGINAL RUBRIC + HI (ORIGINAL RUBRIC + 

ORIGINAL QUESTIONS) ... NE-rf-STYLE QUESTIONS) 

II (NEW RUBRIC + IV (NEW RUBRIC + 

^ nBTr.T»AT. QtiRSTlONS^ ii NEW-STYLE QUESTIONS) , J 



that when new-style questions were involved, the differences in achievement 
engendered .by .the different rubrics (in favour of the original rubric) were 
clearly reversed. Although the difference was not statistically significant, 
the mean achievement score of candidates taking .the paper including the new 
rubric (Paper IV) was between 2 and 3 marks better ..than .>that for those taking 
the paper with the original rubric (Paper Ili). 

The full force of this finding is clear when one considers that the average 
superiority of the original rubric, when combined with. the original questions, 
was about h.25 marks. For one of the two schools involved in the experiment, 
the mean score for the paper with the original rubric (Paper l) was 
approximately 2.5 marks better than that for the paper with the new rubric 
(Paper II ), when original questions were involved. When, however, new 
questions were involved, the paper with the new rubric (Paper IV) produced 
scores 2.2 marks better than the old rubric paper (ill), an overall change of 
about U.T marks (2.5+2.2). For the other school, the overall change in the 
diiference was about 8.2 marks (5.75 + 2.^+5).* 

Although these figures themselves cannot be taken as Gono^u8^ve, the inference 
would seem to be that although the original rubric was clearly superior in the 
familiar situation, when the situation was less familiar (with the introduction 
of a new format of question) the new rubric began to show its worth. It is also 
interesting to note (comparison (d)) that statistically significant differences 
in favour of the papers with ORIGINAL QUESTIONS changed to non-significant 
differences when the comparison involved the NEW RUBRIC. 

Tests of Readability also showed the new rubric to be less complex than the 
original one, and to be suitable for children with a lower reading age. The 
concept of Readability of a passage or text centres on the interest and 
motivation it generates, its legibility, and the complexity of its words and 
sentences .in relation to the reader's reading ability. Many definitions of 
Readability have been devised. Perhaps Harrison's summary (197*+) is one of 



ERJC « It should perhaps be pointed out that these figures are not inconsiderable, 

given the overall size of the mean scores involved: • see Appendix 2(e). yj^ 



the most helpful: 

Broadly speaking readability studies are concerned with the^ 

features of a printed text which tend to make it easier or more 
difficult to read and understand. These features can best be seen 
in terms of four groups: content, format, organization and style. 

In the quotation above, content refers to subject-matter, format to 

legibility and illustrations etc., organisation to use of paragraphs and 

sequencing of ideas, and style to vocabulary and the complexity of sentences. 

There are a considerable number of procedures and formulae for assessing the 

19 

reada:bility of texts in practical terms, so that one can begin to 
establish whether a given text will be suitable for a child at a given stage 
in his school career. The lower the 'readii*g age' needed to read and 
understand a passage, the easier that passage is deemed to be. 

20 

A very similar type of measure, known as Syntactic Density, was also used 
to compare the new and old rubrics. The concept of Syntactic Density is 
linked to Readability, and is based upon the idea that the more complex the 
sentence structure of a passage, the harder it will be to understand. These 
tests again showed that the new rubric was simpler and easier to understand. 

Taken together, the findings concerning the achievement results and those 
concerning the Readability of the rubrics suggest that it is reasonable to put 
forward the idea that although the new rubric may not actually have been 
clearly superior to the original one, it might well, as a factor in the 
examination situation, produce favourable results. Furthermore, the fact that 
the students were well-acquainted with the all-original Paper I, may well have 
been an important factor in its success. Over the course of time during which 
candidates were being prepared for the public examination (of which Paper I 
forms a normal part) the rubric had no doubt been 'translated' and explained 
to the pupils by their teacher. By contrast, the new rubric would have been 
^ completely unfamiliar to them, and had to stand or fall by its own merits 

alone. / 3 
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Perhaps one should also hear in mind that Hamtleton and Trauh report that 
Sarason"'"'* only found differences in performance, resulting from the 
different instructions given, when the conditions were stressful. Smith- and 
Rockett''"^ also found that differences in instructions had no effect on 
performance when the subjects were not highly anxious. The candidates who 
took part in the present experiment were not told that they were testing out 
the ruhrics and question-styles of the different papers until after the 
examination, hut they may not have heen under sufficient stress for 
achievement differences to reflect the real differences hetween the rubrics. 

Analysis of candidates' choice of questions also produced some interesting 
findings. Again, although the new question-styles did not affect performance 
scores materially, they did influence candidates' choice of question. When 
the original straightforward slot-filling technique of one question was 
removed in favour of the new standard format, the question's popularity 
declined. On the other hand, when the original complex map format of another 
question was replaced by the new format, that question became more popular. 
These findings lead one to believe that the matter of question-style is an 
imrortant one, and may indeed outweigh the actual subject content in the 
candidate's process of choice. 

This analysis also showed that, for all four papers, higher percentages of 
candidates tended to choose the questions appearing early in Part One rather 
than the ones which came later, which was also true of the questions at the 
beginning of Part Two of the paper. This finding may perhaps relate to 
Francis and Owen's categorization of examinees' strategies, "^"^ particularly 
the 'Rusher' strategy, which involves answering first the first question 
encountered that one is able to answer. The first three questions on the 
paper may have tended to be so popular for this reason. 

With hindsight, one realises that an attitude questionnaire administered 
to the candidates immediately after the examination might have yielded 
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considerable information about the psychological effects the rubrics and 
question-styles may have had. The writer strongly suspects that such a. 
questionnaire might well have disclosed that candidates felt somewhat less 
stressed by the slightly less fonnal instructions and the more informative 

rubric (as work by Shapland^^ would suggest). If the less complex new 

rubric did reduce stress, then that in itself would have been a valuable 
outcome, even if actual performance were not greatly altered in the short 
term. Indeed, the reduction of stress may actually aid performance in the 
long r\in. 

To summarize, the results of this experiment have shown that the original 
paper produced better achievement scores in straight comparisons with the 
revised versions. However, closer study revealed use of the new rubric to be 
associated with superior performance when connected with the new-style 
questions in a more unfamiliar situation. Readability and Syntactic Density 
tests also showed the new rubric to be less complex than the original one, and 
to be suitable for children with a lower reading age. Although the new 
question-styles did not affect performance scores materially, they did 
influence candidates' choice of questions. 

Suggestions for the Future 

What implications do these findings hold for the constructors of 
examinations? Firstly, as many previous writers have pointed out, if we are 
going to produce papers which permit choice and various different 
combinations of questions, then we must ensure that the style and difficulty 
of the questions are comparable. 

As our recent experiment demon s^-^r at ed, the way in which the question was 
couched had an important influence on candidates' choice of questions, 
irrespective of the question subject-matter. We must surely be careful not 
to dissuade a candidate from a question, to which he knows the answer, by the 

IS 



appearance of that question. 

Secondly, if there is to be choice, then the instructions for that choice 

must be clear. The rubric should not provide a hurdle for the candidate to 

survive before he can begin to answer the questions which form the point of 

4.10 

the examination paper. We must not forget Willmott and Hall's comment 
about a particular examination being easy compared with its rubric. 

Perhaps it will not be practicable, or even desirable, to submit to 
Readability testing every rubric and every question for every examination that 
is written in the future, in order to assess its suitability. However, it 
would seem to make good sense for us to apply the techniques for producing 
•readable' material to the preparation of examinatipn papers. 

Clearly, it is important to think of the 'interest and motivation .... 
legibility .... complexity'^2 of the material, and to consider the 'content, 
format, organization and style'^^ of the. examination papers that are prepared 
for candidates. It is surely essential that papers be constructed for pupils 
in such a way that they will be able to understand, without difficulty, what 
is required of them. Even if simplified rubrics and more systematically 
organised questions did not, in the case of this experiment, influence 
performance significantly, one would suggest that they are likely to be 
valuable in reducing the stress of examinations on candidates, and may in the 
long run aid performance. 

It is perhaps the results of the experiment concerned with the Readability of 
the examination instructions which hold the most significance for those who 
design examination papers, as the options for study in school increase, and 
the consequent choice in papers becomes wider. In Britain especially, with 
the proposed change in emphasis from norm-referenced to criterion-referenced 
examinations, less able candidates are likely to have a greater opportunity tc 
display their knowledge. It will therefore become particularly important 



for them to be given examination instructions which are clear, 
straightforward, simple, and suitable for those with lower reading ages. 

The task of the examination constructor is a very skilled one. 
necessitating as it does, not only the drafting of questions, but adherence 
to the constraints imposed by needing to cover the syllabus fairly and to 
produce original papers year by year. In presenting these findings one 
would not wish to increase the difficulty of that task. 

One would however like to put forward the following overall suggestion: 
that the needs of the candidate, and the possible effects of the content and 
structure of the paper upon him, be considered even more actively in the 
drafting and' monitoring of examination papers - in all subjects and at all 
levels. 

Dr. Elaine S. Freedman 

University of Leicester School of Education 
July 1982. 
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PART ONE 



you must answer SIX questions from this part. 



I, (a) From the choice given, 



write out 



SECTION A 

the word or phrase which correctly completes the foUowing statements: 
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(i) One cause of Anglo-German tension before 1914 was 
the German seizure of Alsace-Lorraine, 
the building of the Kiel Canal, 
England's decision to end her alliance with Germany. 
Germany's building of 'pocket' battleships, 
the tactlessness of Kaiser WiUiam I. 
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You are required to answer SIX questions out of the TWEL VE in FAR T ONE of the examination paper 
(Questions I • 12), and THREE out of the SIX in FART TWO of the paper /Questions 13 • IB). 

The numbers of the questions you have answered should be ticked in the grid above. 

Credit will be given for sketchmaps, diagrams and other illustrations where they are appropriate. 

You must cancel clearly any work you do not wish to be marked. 

All the questions in Part One are worth 10 marks each and aU the questions in Fart Two are worth 20 marks each. 

To help you decide which questions to answer in FAR T ONE, here is a list of the questions in that part of the paper. 
It shows each question number and beside it the topic covered by that question. 
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SECTION D 


QUESTION 


10 




QUESTION 
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QUESTION 


12 



The First World War. 

Creation of an International Organisation. 
First World War Peace COhferences, 

The Spanish Civil War. 
The Rise of Nasi Germany. 
America between the World Wart, 

The Second World War, 
Second World War Conferences. 
The United Nations. 

China. 

The Cold War. 

The European Movement* 
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12. (a) The Prime Minister who tkri look Britain into the E.E.C. was 

Harold Wilson. 
Edward Heath. : 
Lord Home. 
Anthony Eden. 
Harold Macmillan. 



.(1 mark) 



(b) The European Statesman who tried to prerent Britain's entry into the EJE.C. was 



Adenauer. 
Schmidt. 
Erhardt. 
Monnet. 
de Gaulle. 



.(1 mark) 



(c) What is E.F.T.A.? 



(2 marks) 



(d) Why did some British statesmen oppose Britain's entry into the E.E.C.? 



(2 marks) 



(e) Write a paraeraph ( about four sentences) about the E.E.C. 



(4 marks) 
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APPENDIX 2 

RESULTS OF PAPER COMPARISO.NS (PART ONE TOTALS ONLY) 





PAPERS 


FIRST EXAMINER MARKS 


SECOND EXAMINER MARKS 


COMPARISON 




RTRNTFICANCE 


MEAN SCORE 


SIGNIFICANCE 


a) Rubrics 
I & II 


I 0 
II U 


26.0000 

21.7U07 


1 P = O.OU9 


26.0577 
21.7037 




p = O.OU5 


III & IV 


lllO 
IV ii? 


19.3019 
21.7925 


1 Not ';ig. 


19.7358 
2I.8U9I 




Not sig. 


b) Questions 
I & III 


I 0 
lllN 


26.0000 
19.3019 


1 p = 0.002 


26.0577 
19.7358 




: p = O.OOU 


II & IV 


II 0 
IV 


21.7U07 
21.7925 


1 Not sig. 


21.7037 
2I.8U9I 




' Not sig. 


c) R & Q 
combined 
I & IV 


I 0 
IV ^ 


26.0000 
21.7925 


1 Not sig. 


28.1923 
22.9615 




i Not sig. 



WHERE TOTAL POSSIBLE SCORE = 60 

COMPARISON d) 

The effect of the different rubrics on the effect of question sty le 



0 = ORIGINAL 

N = 



Comparison of 'I and III' vith '11 and IV' (b) shov3, for both schools in the 
experiment, significant differences in favour of paper with original questions 
changing to non-significant differences vhen comparison involved nev rubric. 



BOTH WITH 
ORIGINAL 
RUBRIC 



BOTH WITH 

NEW 
RUBRIC 
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SCHOOL 


PAPERS 


FIRST EXAMINER MARKS 


SECOND EXAMINER MARKS 


MEAN SCORE 


SIGNIFICANCE 


MEAN SCORE 


SIGNIFICANCE 


No. 1 


I OQ 
111 NQ 


28.0769 
20.6538 




1 p = 0.036 


28.1923 
2O.8U62 


• 


: p = 0.038 


No. 2 


1 OQ 
111 NQ 


23.9231 
18.0000 




1 p = 0.020 


23.9231 
18.6667 




: p = 0.037 


No. 1 


11 OQ 
IV NQ 


25.U23I 
.22.88U6 


■ 


'. Not sig. 


25.6923 
22.9615 




1 Not sig. 


No. 2 


11 OQ 
IV NQ 


18.0357 
20.7U07 




1 Not sig. 


18.2857 
20.7778 


j Not sig. 



PART ONE TOTAL POSSIBLE SCORE = 60 



COMPARISON e ) 

The effect of the different question-style s on the effect. of rubric 

Comparison of 'I and II' with 'III and IV' (a) shows significant differences 
in favour of paper with original rubric changing to non-significant 
differences when comparison involved new-style questions. In terms of actual 
figures, however, the differences in achievement engendered by the differenr 
rubrics (in favour of the original rubric) are clearly reversed for both 
schools' data. 



BOTH WITH 
ORIGINAL 
QUESTIONS 



BOTH WITH 
NEW-STYLE 
QUESTIONS 



SCHOOL 


PAPERS ' 


FIRST EXAMINER MARKS 


SECOND EXAMINER MARKS 


MEAN SCORE 


SIGNIFICANCE 


MEAN SCORE 


SIGNIFICANCE 


No.. 1 


I OH 
II NH 


28.0769 
25.^231 




Not sig. 


28.1923 
25.6923 


i 


Not sig. 


No. 2 


I OH 
11 NH 


23.9231 
18.0357 




' p = 0.020 


23.9231 
18.2857 




' p = 0.025 


No. 1 


III OH 

IV NH 


20.6538 
22.881+6 . 




[ Not sig. 


20.81+62 
.22.9615 . 


1 Not sig. 


No. 2 


III OH 

IV NH 


18.0000 
20.71+07 




I Not- sig. 


18.6667 
20.7778 




L Not sig. 



PART ONE TOTAL POSSIBLE SCORE = 60 

OR = ORIGINAL RUBRIC 
= NEW RUBRIC 



N.B. The figures quoted in the text, concerning the effect of the 
different question-styles on the effect of the rubric at the 
two different schools, are based on averages of the two 
examiners' marks for each school. 
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